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ABSTRACT 



In 1999 a validation study of the 1998 Civics Assessment of 
the National Assessment of Educational Progress (NAEP) was conducted, 
combining a similarity classification study and a booklet classification 
study. The rationale was that if the teachers who had participated in the 
5-day achievement levels-setting process could not use the descriptions to 
judge student performance, it is unlikely that anyone could. If their 
classifications were very different from the performances in the test 
booklets, it would suggest that the cutpoints did not denote performance 
consistent with the achievement level descriptions (ALDs) . Eleven 
eighth-grade teachers from the pilot study of achievement level setting panel 
participated. Teacher panelists tended to classify their own students higher 
than their performance levels on the special form of the NAEP developed for 
this study. When the same teachers were asked to classify the performance of 
students represented in the special Civics NAEP test booklets, they tended to 
classify those at or below the empirical score classification of the 
students' performance. These findings suggest that standards set with a 
booklet classification method will be higher than those set with the 
item-by-item method used for the NAEP achievement level setting process. Even 
teachers well trained in the NAEP achievement levels tend to overestimate the 
knowledge and skills of their students with respect to the ALDs. Overall, 
however, results provide information needed to confirm that the general 
achievement levels-setting process appeared to "work" in that panelists were 
able to use the ALDs in a different setting and for different purposes and 
the "translations" with respect to the score scale seem reasonably on target. 
An appendix contains classification forms from the similarities 
classification study, a list of participants in the validation portion of the 
study, and an agenda for the study session. (Contains 14 references.) (SLD) 
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